You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "PavithranRick (via GitHub)" <gi...@apache.org> on 2025/12/05 01:16:54 UTC
[PR] HUDI-17479 - Skip expression index for direct column access queries [hudi]
PavithranRick opened a new pull request, #17489:
URL: https://github.com/apache/hudi/pull/17489
### Describe the issue this Pull Request addresses
This PR fixes incorrect matching behavior in expression index pruning.
Currently, queries that reference the raw data column (e.g., `ts`) are incorrectly
matched against expression indexes (e.g., `from_unixtime(ts)`), causing casting
exceptions during pruning. Although the engine swallows the errors and continues,
it still performs unnecessary lookups.
This PR ensures expression indexes are matched **only when the query expression
uses the same function** as the index definition. Direct column references skip
expression-index matching (unless the index function is `identity`). This prevents
invalid matches and avoids inefficient or erroneous pruning attempts.
### Summary and Changelog
- Added function-type validation in `extractQueryAndLiterals` to ensure query
expressions match the index's expression function.
- Direct column references no longer match expression indexes.
- Updated logic around line 443 of `ExpressionIndexSupport.scala`.
- Eliminates silent casting failures caused by mismatched expression types.
- No public API changes; logic fix only.
### Impact
- Improved correctness: non-matching predicates (e.g., `ts = value`) will no
longer attempt expression index pruning such as `from_unixtime(ts)`.
- Reduced overhead: avoids unnecessary failed lookups and swallowed exceptions.
- More predictable and safe index pruning behavior.
### Risk Level
Low
Change is isolated to expression-index match logic.
Verification included:
- Tests for matching-with-function predicates.
- Tests for non-matching direct column predicates.
- Tests for mismatched expression types.
### Documentation Update
None
No new configs or user-facing features. Existing documentation remains valid.
### Contributor's checklist
- [ ] Read through contributor's guide
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: HUDI-17479 - Skip expression index for direct column access queries [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17489:
URL: https://github.com/apache/hudi/pull/17489#issuecomment-3618409153
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9929",
"triggerID" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"triggerType" : "PUSH"
}, {
"hash" : "205226d70469c596c2af2e2bb2731728d9b37121",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9934",
"triggerID" : "205226d70469c596c2af2e2bb2731728d9b37121",
"triggerType" : "PUSH"
}, {
"hash" : "007b1bedc02b9cc59df4aace8b62abac22703b17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9956",
"triggerID" : "007b1bedc02b9cc59df4aace8b62abac22703b17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 007b1bedc02b9cc59df4aace8b62abac22703b17 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9956)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HUDI-17479 - Skip expression index for direct column access queries [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17489:
URL: https://github.com/apache/hudi/pull/17489#issuecomment-3614933920
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe9bcf269839ac3717311a9f3c6b2797f509ed3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: HUDI-17479 - Skip expression index for direct column access queries [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17489:
URL: https://github.com/apache/hudi/pull/17489#issuecomment-3615047709
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9929",
"triggerID" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"triggerType" : "PUSH"
}, {
"hash" : "205226d70469c596c2af2e2bb2731728d9b37121",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9934",
"triggerID" : "205226d70469c596c2af2e2bb2731728d9b37121",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe9bcf269839ac3717311a9f3c6b2797f509ed3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9929)
* 205226d70469c596c2af2e2bb2731728d9b37121 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9934)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: HUDI-17479 - Skip expression index for direct column access queries [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17489:
URL: https://github.com/apache/hudi/pull/17489#issuecomment-3618310169
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9929",
"triggerID" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"triggerType" : "PUSH"
}, {
"hash" : "205226d70469c596c2af2e2bb2731728d9b37121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9934",
"triggerID" : "205226d70469c596c2af2e2bb2731728d9b37121",
"triggerType" : "PUSH"
}, {
"hash" : "007b1bedc02b9cc59df4aace8b62abac22703b17",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9956",
"triggerID" : "007b1bedc02b9cc59df4aace8b62abac22703b17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 205226d70469c596c2af2e2bb2731728d9b37121 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9934)
* 007b1bedc02b9cc59df4aace8b62abac22703b17 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9956)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: HUDI-17479 - Skip expression index for direct column access queries [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17489:
URL: https://github.com/apache/hudi/pull/17489#issuecomment-3614937441
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9929",
"triggerID" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe9bcf269839ac3717311a9f3c6b2797f509ed3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9929)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: HUDI-17479 - Skip expression index for direct column access queries [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17489:
URL: https://github.com/apache/hudi/pull/17489#issuecomment-3618302150
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9929",
"triggerID" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"triggerType" : "PUSH"
}, {
"hash" : "205226d70469c596c2af2e2bb2731728d9b37121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9934",
"triggerID" : "205226d70469c596c2af2e2bb2731728d9b37121",
"triggerType" : "PUSH"
}, {
"hash" : "007b1bedc02b9cc59df4aace8b62abac22703b17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "007b1bedc02b9cc59df4aace8b62abac22703b17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 205226d70469c596c2af2e2bb2731728d9b37121 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9934)
* 007b1bedc02b9cc59df4aace8b62abac22703b17 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: HUDI-17479 - Skip expression index for direct column access queries [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17489:
URL: https://github.com/apache/hudi/pull/17489#issuecomment-3614991279
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9929",
"triggerID" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"triggerType" : "PUSH"
}, {
"hash" : "205226d70469c596c2af2e2bb2731728d9b37121",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "205226d70469c596c2af2e2bb2731728d9b37121",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe9bcf269839ac3717311a9f3c6b2797f509ed3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9929)
* 205226d70469c596c2af2e2bb2731728d9b37121 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: HUDI-17479 - Skip expression index for direct column access queries [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #17489: fix: HUDI-17479 - Skip expression index for direct column access queries
URL: https://github.com/apache/hudi/pull/17489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: HUDI-17479 - Skip expression index for direct column access queries [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #17489:
URL: https://github.com/apache/hudi/pull/17489#issuecomment-3615100222
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9929",
"triggerID" : "ffe9bcf269839ac3717311a9f3c6b2797f509ed3",
"triggerType" : "PUSH"
}, {
"hash" : "205226d70469c596c2af2e2bb2731728d9b37121",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9934",
"triggerID" : "205226d70469c596c2af2e2bb2731728d9b37121",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 205226d70469c596c2af2e2bb2731728d9b37121 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9934)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: HUDI-17479 - Skip expression index for direct column access queries [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #17489:
URL: https://github.com/apache/hudi/pull/17489#issuecomment-6700864646
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org