You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/09/20 15:04:18 UTC
[PR] fix(ingest): HudiIncrSource: Handle nullable partition filter, use getters and make instance variables private [hudi]
the-other-tim-brown opened a new pull request, #13945:
URL: https://github.com/apache/hudi/pull/13945
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
The `SnapshotLoadQuerySplitter` can return a null partition filter which results in an error thrown at runtime.
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
- Switches from `Option.of` to `Option.ofNullable` to handle the possibly null filter string.
- Updates the `SnapshotLoadQuerySplitter` to have private and final instance variables are rely on getter methods as accessors for these fields.
### Impact
Fixes a bug for HoodieStreamer users
### Risk Level
None
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ingest): HudiIncrSource - Handle nullable partition filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13945:
URL: https://github.com/apache/hudi/pull/13945#issuecomment-3315262573
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8671",
"triggerID" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"triggerType" : "PUSH"
}, {
"hash" : "2e3c09cd598031032150e2f8e7fec3ab6731c860",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8674",
"triggerID" : "2e3c09cd598031032150e2f8e7fec3ab6731c860",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e3c09cd598031032150e2f8e7fec3ab6731c860 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8674)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ingest): HudiIncrSource - Handle nullable partition filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13945:
URL: https://github.com/apache/hudi/pull/13945#issuecomment-3315199799
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8671",
"triggerID" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"triggerType" : "PUSH"
}, {
"hash" : "2e3c09cd598031032150e2f8e7fec3ab6731c860",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2e3c09cd598031032150e2f8e7fec3ab6731c860",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 68d9f1dd8f68699b5426aa8dda9a83720edaa01c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8671)
* 2e3c09cd598031032150e2f8e7fec3ab6731c860 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ingest): HudiIncrSource - Handle nullable partition filter [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #13945:
URL: https://github.com/apache/hudi/pull/13945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ingest): HudiIncrSource - Handle nullable partition filter [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13945:
URL: https://github.com/apache/hudi/pull/13945#discussion_r2365773595
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -269,11 +269,11 @@ private Pair<Option<Dataset<Row>>, Checkpoint> fetchNextBatchBasedOnCompletionTi
Option<SnapshotLoadQuerySplitter.CheckpointWithPredicates> newCheckpointAndPredicate =
snapshotLoadQuerySplitter.get().getNextCheckpointWithPredicates(snapshot, queryContext);
if (newCheckpointAndPredicate.isPresent()) {
- endCompletionTime = newCheckpointAndPredicate.get().endCompletionTime;
- predicate = Option.of(newCheckpointAndPredicate.get().predicateFilter);
+ endCompletionTime = newCheckpointAndPredicate.get().getEndCompletionTime();
+ predicate = Option.ofNullable(newCheckpointAndPredicate.get().getPredicateFilter());
Review Comment:
Possible to add a unit test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ingest): HudiIncrSource - Handle nullable partition filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13945:
URL: https://github.com/apache/hudi/pull/13945#issuecomment-3315086112
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8671",
"triggerID" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 68d9f1dd8f68699b5426aa8dda9a83720edaa01c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8671)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ingest): HudiIncrSource - Handle nullable partition filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13945:
URL: https://github.com/apache/hudi/pull/13945#issuecomment-3315200389
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8671",
"triggerID" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"triggerType" : "PUSH"
}, {
"hash" : "2e3c09cd598031032150e2f8e7fec3ab6731c860",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8674",
"triggerID" : "2e3c09cd598031032150e2f8e7fec3ab6731c860",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 68d9f1dd8f68699b5426aa8dda9a83720edaa01c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8671)
* 2e3c09cd598031032150e2f8e7fec3ab6731c860 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8674)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ingest): HudiIncrSource - Handle nullable partition filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13945:
URL: https://github.com/apache/hudi/pull/13945#issuecomment-3315040134
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8671",
"triggerID" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 68d9f1dd8f68699b5426aa8dda9a83720edaa01c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8671)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ingest): HudiIncrSource - Handle nullable partition filter [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13945:
URL: https://github.com/apache/hudi/pull/13945#issuecomment-3315039174
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "68d9f1dd8f68699b5426aa8dda9a83720edaa01c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 68d9f1dd8f68699b5426aa8dda9a83720edaa01c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ingest): HudiIncrSource - Handle nullable partition filter [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13945:
URL: https://github.com/apache/hudi/pull/13945#discussion_r2365782490
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -269,11 +269,11 @@ private Pair<Option<Dataset<Row>>, Checkpoint> fetchNextBatchBasedOnCompletionTi
Option<SnapshotLoadQuerySplitter.CheckpointWithPredicates> newCheckpointAndPredicate =
snapshotLoadQuerySplitter.get().getNextCheckpointWithPredicates(snapshot, queryContext);
if (newCheckpointAndPredicate.isPresent()) {
- endCompletionTime = newCheckpointAndPredicate.get().endCompletionTime;
- predicate = Option.of(newCheckpointAndPredicate.get().predicateFilter);
+ endCompletionTime = newCheckpointAndPredicate.get().getEndCompletionTime();
+ predicate = Option.ofNullable(newCheckpointAndPredicate.get().getPredicateFilter());
Review Comment:
Updated the PR with a test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org