You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2024/09/20 09:01:06 UTC
[PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
codope opened a new pull request, #11975:
URL: https://github.com/apache/hudi/pull/11975
### Change Logs
Suppose RLI (record index) is enabled on MOR table, do an insert and then delete a record. When table is not compacted, it is expected that RO (read-optimized) query shows the deleted record, while snapshot query will not. A query with no predicates works fine, but if the query has a predicate on record key field, it will use RLI and RLI always returns the latest snapshot. Hence, this PR fixes the issue by not using RLI for queries other than snapshot queries. The PR adds a new `supportsQueryType(queryType: HoodieTableQueryType)` API in the base class for index support. Different index can implement is differently as and when we add time tavel support for indexes.
### Impact
Fix RO queries with RLI and record key predicate.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2363426541
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626",
"triggerID" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"triggerType" : "PUSH"
}, {
"hash" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31836f85df3ee8d56902fbb7b393d8ca5e6d7ada Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626)
* 0c0b22a85163177da915031e39742f2cc73d08e5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2371889477
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 518f9d4889b898cf8621a3dc4d7ac83fad9a825b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2373197032
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744",
"triggerID" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"triggerType" : "PUSH"
}, {
"hash" : "ea5580f728e97b7e594a6d163e3f9badeccc5a38",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=770",
"triggerID" : "ea5580f728e97b7e594a6d163e3f9badeccc5a38",
"triggerType" : "PUSH"
}, {
"hash" : "5901ebbd54ddd2f639d6035d329378b42020e136",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5901ebbd54ddd2f639d6035d329378b42020e136",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 518f9d4889b898cf8621a3dc4d7ac83fad9a825b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744)
* ea5580f728e97b7e594a6d163e3f9badeccc5a38 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=770)
* 5901ebbd54ddd2f639d6035d329378b42020e136 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2367215318
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=662",
"triggerID" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"triggerType" : "PUSH"
}, {
"hash" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c16f9ec0b427a73f849eb33f59d506c57610d721 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=662)
* 0ac120115fa08da70c857ef4032ae35785f08c6e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2365138790
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c16f9ec0b427a73f849eb33f59d506c57610d721 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2366729381
> Then can we just handle this logic in RLI index because the issue is specific to it? Just skip the data skipping inside RLI index should be working right? We can introduce the new API if more index type are involved I think.
It's for RLI and Secondary index. I can remove the colstats override. The default implementation of API returns true which should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2373075379
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744",
"triggerID" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"triggerType" : "PUSH"
}, {
"hash" : "ea5580f728e97b7e594a6d163e3f9badeccc5a38",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ea5580f728e97b7e594a6d163e3f9badeccc5a38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 518f9d4889b898cf8621a3dc4d7ac83fad9a825b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744)
* ea5580f728e97b7e594a6d163e3f9badeccc5a38 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11975:
URL: https://github.com/apache/hudi/pull/11975#discussion_r1774661761
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -91,6 +93,23 @@ class RecordLevelIndexSupport(spark: SparkSession,
def isIndexAvailable: Boolean = {
metadataConfig.isEnabled && metaClient.getTableConfig.getMetadataPartitions.contains(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX)
}
+
+ /**
+ * Returns true if the query type is supported by the index.
+ */
+ override def supportsQueryType(options: Map[String, String]): Boolean = {
+ options.get(TIME_TRAVEL_AS_OF_INSTANT.key)
+ .fold {
+ // if the as.of.instant is None, check QUERY_TYPE is snapshot
+ options.getOrElse(QUERY_TYPE.key, QUERY_TYPE.defaultValue).equalsIgnoreCase(HoodieTableQueryType.SNAPSHOT.name)
+ false
Review Comment:
good catch, removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2365440071
Then can we just handle this logic in RLI index because the issue is specific to it? Just skip the data skipping inside RLI index should be working right? We can introduce the new API if more index type are involved I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2367281490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=662",
"triggerID" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"triggerType" : "PUSH"
}, {
"hash" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=710",
"triggerID" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ac120115fa08da70c857ef4032ae35785f08c6e Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=710)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2371892060
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744",
"triggerID" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 518f9d4889b898cf8621a3dc4d7ac83fad9a825b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2365024079
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626",
"triggerID" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"triggerType" : "PUSH"
}, {
"hash" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=629",
"triggerID" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"triggerType" : "PUSH"
}, {
"hash" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c0b22a85163177da915031e39742f2cc73d08e5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=629)
* c16f9ec0b427a73f849eb33f59d506c57610d721 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2365108280
> A query with no predicates works fine, but if the query has a predicate on record key field, it will use RLI and RLI always returns the latest snapshot. Hence, this PR fixes the issue by not using RLI
Can we just skip the file filtering if we find the file in the index item is a log file when rdo RO query, so that we can eliminate the new API on the file index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2373205124
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744",
"triggerID" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"triggerType" : "PUSH"
}, {
"hash" : "ea5580f728e97b7e594a6d163e3f9badeccc5a38",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=770",
"triggerID" : "ea5580f728e97b7e594a6d163e3f9badeccc5a38",
"triggerType" : "PUSH"
}, {
"hash" : "5901ebbd54ddd2f639d6035d329378b42020e136",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=774",
"triggerID" : "5901ebbd54ddd2f639d6035d329378b42020e136",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ea5580f728e97b7e594a6d163e3f9badeccc5a38 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=770)
* 5901ebbd54ddd2f639d6035d329378b42020e136 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=774)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2367506465
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=710",
"triggerID" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ac120115fa08da70c857ef4032ae35785f08c6e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=710)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2365139258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=662",
"triggerID" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c16f9ec0b427a73f849eb33f59d506c57610d721 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=662)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11975:
URL: https://github.com/apache/hudi/pull/11975#discussion_r1774662103
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -91,6 +93,23 @@ class RecordLevelIndexSupport(spark: SparkSession,
def isIndexAvailable: Boolean = {
metadataConfig.isEnabled && metaClient.getTableConfig.getMetadataPartitions.contains(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX)
}
+
+ /**
+ * Returns true if the query type is supported by the index.
+ */
+ override def supportsQueryType(options: Map[String, String]): Boolean = {
+ options.get(TIME_TRAVEL_AS_OF_INSTANT.key)
Review Comment:
makes sense, updated the condition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2371475969
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=710",
"triggerID" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"triggerType" : "PUSH"
}, {
"hash" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744",
"triggerID" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ac120115fa08da70c857ef4032ae35785f08c6e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=710)
* 518f9d4889b898cf8621a3dc4d7ac83fad9a825b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2367503867
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ac120115fa08da70c857ef4032ae35785f08c6e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2363428884
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626",
"triggerID" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"triggerType" : "PUSH"
}, {
"hash" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=629",
"triggerID" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31836f85df3ee8d56902fbb7b393d8ca5e6d7ada Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626)
* 0c0b22a85163177da915031e39742f2cc73d08e5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=629)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #11975:
URL: https://github.com/apache/hudi/pull/11975#discussion_r1774663452
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala:
##########
@@ -1418,4 +1417,81 @@ class TestMORDataSource extends HoodieSparkClientTestBase with SparkDatasetMixin
metaClient = createMetaClient(spark, basePath)
assertEquals(metaClient.getTableConfig.getRecordMergerStrategy, mergerStrategyName)
}
+
+ /**
+ * Test Read-Optimized query on MOR table with RECORD_INDEX enabled.
+ */
+ @Test
+ def testReadOptimizedQueryWithRecordIndex(): Unit = {
Review Comment:
renamed to better reflect the test purpose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2371673485
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=710",
"triggerID" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"triggerType" : "PUSH"
}, {
"hash" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744",
"triggerID" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 518f9d4889b898cf8621a3dc4d7ac83fad9a825b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2371471210
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=710",
"triggerID" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"triggerType" : "PUSH"
}, {
"hash" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0ac120115fa08da70c857ef4032ae35785f08c6e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=710)
* 518f9d4889b898cf8621a3dc4d7ac83fad9a825b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11975:
URL: https://github.com/apache/hudi/pull/11975#discussion_r1771217900
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkBaseIndexSupport.scala:
##########
@@ -45,6 +44,8 @@ abstract class SparkBaseIndexSupport(spark: SparkSession,
def isIndexAvailable: Boolean
+ def supportsQueryType(queryType: HoodieTableQueryType): Boolean = true
Review Comment:
NIT: We can add javadocs for this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2367216953
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=662",
"triggerID" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"triggerType" : "PUSH"
}, {
"hash" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=710",
"triggerID" : "0ac120115fa08da70c857ef4032ae35785f08c6e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c16f9ec0b427a73f849eb33f59d506c57610d721 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=662)
* 0ac120115fa08da70c857ef4032ae35785f08c6e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=710)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2363227914
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626",
"triggerID" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31836f85df3ee8d56902fbb7b393d8ca5e6d7ada Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11975:
URL: https://github.com/apache/hudi/pull/11975#discussion_r1771219310
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkBaseIndexSupport.scala:
##########
@@ -45,6 +44,8 @@ abstract class SparkBaseIndexSupport(spark: SparkSession,
def isIndexAvailable: Boolean
+ def supportsQueryType(queryType: HoodieTableQueryType): Boolean = true
Review Comment:
Can we also add a TODO here that we make the default behaviour to throw an exception once the API is implemented for all the indexes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2365049837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626",
"triggerID" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"triggerType" : "PUSH"
}, {
"hash" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=629",
"triggerID" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"triggerType" : "PUSH"
}, {
"hash" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=662",
"triggerID" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c16f9ec0b427a73f849eb33f59d506c57610d721 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=662)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2363390830
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626",
"triggerID" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31836f85df3ee8d56902fbb7b393d8ca5e6d7ada Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2365024548
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626",
"triggerID" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"triggerType" : "PUSH"
}, {
"hash" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=629",
"triggerID" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"triggerType" : "PUSH"
}, {
"hash" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=662",
"triggerID" : "c16f9ec0b427a73f849eb33f59d506c57610d721",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c0b22a85163177da915031e39742f2cc73d08e5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=629)
* c16f9ec0b427a73f849eb33f59d506c57610d721 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=662)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2365138352
> Can we just skip the file filtering if we find the file in the index item is a log file when rdo RO query, so that we can eliminate the new API on the file index. And a RO query on full compacted table can also utilitize the RLI index.
@danny0405 This is a good point. But, the index item in RLI only contains fileId and not the file name. Moreover, I still think it makes sense to have this new API because think about time travel query. Let's say there is an instant `t` in the past which is a compaction instant and table was fully compacted at that instant. If user runs a query with record key predicate and `as of instant t` then RLI would still return candidate files as per the latest snapshot. And RO queries are nothing but time travel as of the latest compaction time.
I think it's much cleaner to simply not use RLI for any type of queries other than Snapshot queries. Later, when we add support for time travel on metadata table, we can easily change the condition in the implementation of this API and use the index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2373077292
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744",
"triggerID" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"triggerType" : "PUSH"
}, {
"hash" : "ea5580f728e97b7e594a6d163e3f9badeccc5a38",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=770",
"triggerID" : "ea5580f728e97b7e594a6d163e3f9badeccc5a38",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 518f9d4889b898cf8621a3dc4d7ac83fad9a825b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744)
* ea5580f728e97b7e594a6d163e3f9badeccc5a38 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=770)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #11975:
URL: https://github.com/apache/hudi/pull/11975#discussion_r1774570230
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -91,6 +93,23 @@ class RecordLevelIndexSupport(spark: SparkSession,
def isIndexAvailable: Boolean = {
metadataConfig.isEnabled && metaClient.getTableConfig.getMetadataPartitions.contains(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX)
}
+
+ /**
+ * Returns true if the query type is supported by the index.
+ */
+ override def supportsQueryType(options: Map[String, String]): Boolean = {
+ options.get(TIME_TRAVEL_AS_OF_INSTANT.key)
Review Comment:
Should we have the first check on query type? A very rare case might occur where user uses `as.of.instant` with RO query and still allow query to go through.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala:
##########
@@ -1418,4 +1417,81 @@ class TestMORDataSource extends HoodieSparkClientTestBase with SparkDatasetMixin
metaClient = createMetaClient(spark, basePath)
assertEquals(metaClient.getTableConfig.getRecordMergerStrategy, mergerStrategyName)
}
+
+ /**
+ * Test Read-Optimized query on MOR table with RECORD_INDEX enabled.
+ */
+ @Test
+ def testReadOptimizedQueryWithRecordIndex(): Unit = {
Review Comment:
Rename to testQueryTypesWithRLI?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/RecordLevelIndexSupport.scala:
##########
@@ -91,6 +93,23 @@ class RecordLevelIndexSupport(spark: SparkSession,
def isIndexAvailable: Boolean = {
metadataConfig.isEnabled && metaClient.getTableConfig.getMetadataPartitions.contains(HoodieTableMetadataUtil.PARTITION_NAME_RECORD_INDEX)
}
+
+ /**
+ * Returns true if the query type is supported by the index.
+ */
+ override def supportsQueryType(options: Map[String, String]): Boolean = {
+ options.get(TIME_TRAVEL_AS_OF_INSTANT.key)
+ .fold {
+ // if the as.of.instant is None, check QUERY_TYPE is snapshot
+ options.getOrElse(QUERY_TYPE.key, QUERY_TYPE.defaultValue).equalsIgnoreCase(HoodieTableQueryType.SNAPSHOT.name)
+ false
Review Comment:
Unintended change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2373402307
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=744",
"triggerID" : "518f9d4889b898cf8621a3dc4d7ac83fad9a825b",
"triggerType" : "PUSH"
}, {
"hash" : "ea5580f728e97b7e594a6d163e3f9badeccc5a38",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=770",
"triggerID" : "ea5580f728e97b7e594a6d163e3f9badeccc5a38",
"triggerType" : "PUSH"
}, {
"hash" : "5901ebbd54ddd2f639d6035d329378b42020e136",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=774",
"triggerID" : "5901ebbd54ddd2f639d6035d329378b42020e136",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5901ebbd54ddd2f639d6035d329378b42020e136 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=774)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2363225123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 31836f85df3ee8d56902fbb7b393d8ca5e6d7ada UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6891] Fix RO queries with RLI and record key predicate [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11975:
URL: https://github.com/apache/hudi/pull/11975#issuecomment-2363564982
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=626",
"triggerID" : "31836f85df3ee8d56902fbb7b393d8ca5e6d7ada",
"triggerType" : "PUSH"
}, {
"hash" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=629",
"triggerID" : "0c0b22a85163177da915031e39742f2cc73d08e5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0c0b22a85163177da915031e39742f2cc73d08e5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=629)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org